Neural Networks for Mapping Hand Gestures to Sound Synthesis Parameters

This paper expands on our work on mapping hand gestures to musical parameters in an interactive music performance and virtual reality environment 1 . A neural network architecture for subgestures will be introduced to provide a mechanism in order to achieve meaningful control parameters (e.g. aesthetic variations). We use data obtained from a sensor glove, an input device for digitizing hand and finger motions into multi-parametric data. These data are processed in order to extract meaningful data to control musical structures. This is done by an extended neural network architecture for sub-gestures combined with the extraction of parametric values. We focus on the mapping of gestural variations onto equivalent musical parameters which could be used in a performance. We set up a dictionary of symbolic and parametric subgestures. Different hand gestures of this dictionary and characteristic variations will be evaluated with respect to their applicability to intuitive control of musical structures. The system is complemented with a 3D VRML environment, i.e. an animated hand model and behaving representations of musical structures. This 3D representation combines with the gesture processing module and the sound generation engine to produce "Behaving Virtual Musical Objects".